Comments for MEDB 5510, Week 01

Topics to be covered

What you will learn
- Defining Clinical Research Methodology
- Quantitative versus qualitative research
- Other research dichotomies

Learning objectives

To describe the variety of research that can be conducted while doing clinical research.
To describe what is needed in order to identify and define a research question that could be the basis for a research project.

What is research?

” systematic investigation, including research development, testing, and evaluation, designed to develop or contribute to generalizable knowledge”
- 45 CFR 46.102
“Disciplined method” or “disciplined inquiry”
- “dispassionate search for truth”
- not dependent on surface plausibility or the status of the author.
- Gliner, Morgan, Leech. Research Methods in Applied Settings.

Let’s define the three words “clinical research methodology.” Start with “research.” I like the federal government’s definition. This comes from the Code of Federal Regulations. Basically 45 CFR 46 is the section of the Code of Federal Regulations that set up rules and regulations for ethical conduct of research.

The key words in this definition are “generalizable knowledge.” If you don’t hope to extrapolate beyond the range of the data you collect, it isn’t research.

There certain types of studies I like to call “warehouse studies,” which is analogous to trying to find out what is in a certain storage area. Let me give an example from the time I was at Children’s Mercy. They had a complaint about overbilling. In fairness to Children’s Mercy that’s a difficult topic. Quite honestly finding the appropriate level billing is difficult.

There were tens of thousands of records that they had to audit they didn’t want to audit everything. Instead, we created a random sample of the records. The purpose of random samples NOT to generalize to other institutions. The whole idea was to try to find out how much Children’s Mercy overcharged, so we could properly refund the overcharges.

This idea of generalizable knowledge is probably the dividing line between the quality improvement studies and research studies and this is very complicated. I will talk more about the later. We have a whole week dedicated to quality improvement studies.

Gliner, Morgan, and Leech emphasize research as a disciplined method. They emphasize a dispassionate search for truth and independence from preconceived notions. A true researcher should be equally happy if they find that their hypothesis is right or wrong.

Even though internally there may be some level of disappointment, you are happy for your patients. You’re interested not in burnishing your career. You’re interested in finding the truth. So a good researcher rejoices whenever their research hypotheses is disproven because that’s progress. You know more about what and what not to do for your patients. Negative trials are progress just as much as positive trial. I’m sure I’ll talk more about that later. If you have a dispassionate search for the truth, finding out that a certain medication or certain therapy that you think really works did not work is in the best interest of your patients.

Both definitions are good, but I like the government definition for its simplicity.

What is clinical? What is methodology?

Clinical: “concerned with or based on actual observation and treatment of disease in patients rather than experimentation or theory.”
- Dictionary.com
Methodology: “a body of methods, rules, and postulates employed by a discipline : a particular procedure or set of procedures”
- Merriam-Webster dictionary

The classic dictionary definitions for “clinical” and “methodology” work just fine. “Clinical” places the emphasis on patients and methodology is a body of rules.

Now some of you may be interested in non-clinical research, such as animal studies or computer simulations. These types of research will not be covered explicitly in this course, but there is enough commonality that you can still learn a lot of things that will help you in your research career. This also means that all of my examples will be oriented to medicine. There are people who come from outside of the medical field who take this class and they still a lot that can be gotten out of this class. Maybe 75% or more of the content is still relevant, but you just have to live with the fact that all the examples I talked about will be medical examples.

So if you’re in this class and you’re not a clinician, that’s fine. If you animal studies, that’s fine. If you do marketing research that’s fine.

Research “flavors”

Many different nouns, same basic concept
- Research aim
- Research approach
- Research goal
- Research hypothesis
- Research question
- Research topic

I just added this slide because I’ve always had trouble with this. I ask people what you are researching and I don’t know what proper noun is. is it a research aim, research approach, research goal, research hypothesis, research question or research topic. There are slight and subtle differences among these terms, but I use them more or less interchangeably.

A research aim is a highly focused aspect of the research protocol or research grant. It tends to have a very narrow focus.

A research approach is really talking about the methods that you going to use.

A research goal is larger in scope than a research aim.

A research hypothesis is a term that I use more often than anything, but quite honestly a lot of research does not have a hypothesis. We need to get past that belief that if you don’t have a hypothesis, you don’t have research. A research hypothesis is something you postulated based on your understanding of the theory of medicine that you want to prove or disprove by collecting data.

Research hypotheses tend to be black and white and I think sometimes it’s good to think in black-and-white terms. Sometimes, though, trying to choose between only two alternatives is not a good approach to research

I need to warn you about this. If I ask you what you research hypothesis and you have a research questions, don’t go all huffy on me. I use these interchangeably. I shouldn’t but I do. I’ll be honest: I am usually not sloppy with terminology. I am usually very precise with my words but this is one exception.

A research question is a term often used when a research study does not have a hypothesis. Here’s a research question. How do children who have siblings with the chronic disease cope. The child that has a chronic disease get a lot more attention, has a lot of crises that can provoke fear and anxiety in the family, and so forth. There’s no hypothesis here at all. There’s no this mean is bigger than that mean, no this proportion is bigger than that proportion.

Finally a research topic is generally very broad and general.

I hope I got all those distinctions right. I may be wrong on some of the details because I do not fuss about the subtle dstinctions in these terms.

Elements of a research hypothesis

PICO
- P = patient population
- I = intervention (or exposure)
- C = control or comparison group
- O = outcome
PICOTS
- T = time frame
- S = setting

To try to characterize research, especially research that includes a hypothesis, I look for four elements, P, I, C, and O. This comes from evidence-based medicine, the step where you ask an answerable question. In order to do a good job with evidence-based medicine, you first have to pose a question in a way that can be answered by reviewing the literature, reviewing the evidence. Okay, so the experts in evidence based medicine came up with this acronym, PICO. You don’t always apply this too literally, but find that it helps.

P is the patient population: who you studied. Are you studying children, or studying the elderly. Are you studying people who present with the neck fracture? Sometimes it is not people. Sometimes, you are you studying the houses that people live in, because you are interested in environmental hazards like lead paint in the home.

I is the intervention. An intervention is when you when you change something. You try to physically intervene to mitigate the problem. You give a drug or offer a therapy. Sometimes I use the word exposure. Exposure is something that your patients endure. It’s not something that you provide hoping for a cure. Second hand smoke is an exposure. Non-ionizing radiation is an exposure. Interventions are sometimes bad, but the intention is what matter. An intervention is intended to make things better, while an exposure is something that might make things worse.

C is the control or comparison group. Research hypothesis, almost always, involves some sort of comparison groups. Typically it is a control group. The control group either does not get the intervention, or they get part of the intervention only, or they get an alternate intervention.

There certain cases where you don’t have a comparison group. If you are trying to assess the quality of a diagnostic test, it gets a bit harder because everyone gets the diagnostic test. You can try to force this by defining a C, but this is usually not worth the effort.

There’s an extension of this acronym, PICOTS. T specifies the time frame and S specifies the setting. These are important differences between a study based in a tertiary care center rather than in a first-line clinic. I sometimes look for the T and the S, but I always look for the first four, P-I-C-O.

Where do research ideas come from?

Ronan Conroy has an excellent summary.

Exploring your environment.
Don’t focus prematurely on a single idea.
Extending the ideas of others.
Getting a research idea by reading papers.

My theory

Irritation

There’s an excellent summary of how to come up with a research idea that I was asked to host on my website. It was collated from various postings on an email discussion group by Ronan Conroy.

I have lost touch with Ronan Conroy, but he and I use to correspond by Internet a lot. He’s a really smart guy in Ireland who wrote a very nice summary of where do research ideas come from. He pulled a bunch of ideas from the discussion board that he was on and he wrote it up very nicely. He came to me and asked if I could post his summary on my website. I was happy to do this, because it was so well written.

His outline has four main topics. I don’t want to go into too much detail, but the first suggestion is to explore your environment. Look around. Sometimes stuff is sitting right under your nose. Now there are two types of students in this class. There are students who are already deeply involved in clinical responsibilities. They may be a medical resident or fellow, they may actually have the terminal medical degree, and they want to get more involved in research. These students usually are swimming in ideas because there’s all this they see every day in the clinic.

Now I have to say I’m not good at this because I don’t do clinical work. I don’t see patients. I’m looking from the outside. Sometimes that’s a big advantage. I can be more objective about problems, I can look at the problem more abstractly. I can bring in solutions from other disciplines. So it’s good to be an outsider to some extent. But if you’re like me and your background is more mathematical or more computational, you may have trouble coming up with a good research idea. You don’t live in the clinic the way other students do. See if you can talk to someone who has that clinical experience and ask them for help picking a good research topic.

The other thing is to avoid focusing prematurely on a single idea. You need that branch out and not have tunnel vision.

A third approach is to extend the ideas of others. William Shakespeare was a great, great writer, wrote some wonderful plays, but he stole most of these plays from someone else. He just wrote them so much better than the original sources that that’s why we remember his name today and not any of those other people.

Extending the idea of others is not plagiarism. It’s not stealing. It’s part of research. So, for example, there may be a a fair amount known already about how to get people to lose weight when they are physically handicapped and do not have the ability to exercise. Even so, there is always a way to improve one the currently best available interventions.

If we are talking about treatment of cancer, there are always ways we can improve the treatments. For example, you might work to reduce the side effects of some these very harsh drugs.

If you are a beginning resarchers, the last thing you want to try to do is come up with something that’s totally de novo, something that’s brand-new, something that no one else has ever thought of. Few of us, and I put myself in this category, are so smart that they can develop something brand-new is never been thought of before.

The last suggesiton is closely related. Read other people’s research papers. These papers have a literature review, a methods section, a results section, and a discussion section. Look at that discussion and look at where they say “options for further research.” These people are giving you free advice.

Okay, now my theory about research is a lot simpler why people do research. It’s the irritation principal. They see something’s wrong and it just bothers them. There are two types of wrong. There’s I’m doing something right now that I know I could be doing better okay. There is also I’m doing it right, but no one else is. Either way, there’s a sense of irritation. It’s that they have an itch that they just have to scratch. There is something that needs to be fixed and I won’t feel right until I try to fix it. That’s where I think for a lot of research ideas come from.

All these ideas are all good, but let me emphasize the importance of reading. Almost every research paper includes suggestions for further research in their discussion section. This is free advice from researchers who have proved that they can get their work published.

Uses of clinical research

Practical application.
Theory development.
Development of research tools.
Professional development.

Why do we do clinical research? Let’s let’s be honest about this is several reasons. It’s not just the practical application–learning about something that can improve the care you provide to your patients. Sometimes research is done for theory development–understanding why and how diseases progress.

You can also conduct research to develop new tools to improve the research process (this is an area I am very interested in).

You can also conduct research for your own professional development. Nothing makes you a better consumer of other people’s research than doing a bit of research yourself.

It’s important to understand your motivations for doing research, because different motivations result in different priorities. In particular, the pedagogical benefits of conducting a research study will often drive a thesis committee to encourage approaches that broaden a student’s understanding of the research process, even though a different approach might be better.

Don’t get me wrong. Most of the time you can improve your knowledge and benefit your patients at the same time.

Break #1

What you have learned
- Defining Clinical Research Methodology
What’s coming next
- Quantitative versus qualitative research

Quantitative research versus qualitative research

Quantitative research.

Synonym(?): Positivistic.
Highly structured, A priori specifications.
Separates the researcher from the research.
Data is easily represented as numbers

Now this is where you book is often the deep end. They talk about qualitative research versus quantitative research but then they talk about qualitative data versus quantitative data and then they talk about qualitative analysis versus quantitative analysis. I believe these are pretty close to being the same thing. So I’m just going to call it quantitative research versus qualitative research.

I’m not an expert in philosophy, but there is a branch of philosophy called positivism. It closely related to something called empiricism.

Positivism is the belief that if you use a well-defined process, you or anyone else can arrive at what we considered truth.There a lot of people who are very strong proponents of positivism. They believe that there’s a certain procedure that you can follow and if you follow it carefully, researcher bias is removed from the equation. Two researchers, even if one has different preconceived notions than the other will still arrive at the same conclusion.

The Food and Drug Administration (FDA) is probably the regulatory agency that’s furthest along in this idea of positivism. They believe that a company that makes a million-dollar actually close to $1 billion investment in drug development can still produce truthful answers about the safety and efficacy of their drug or device, in spite of all of these economic incentives. The FDA puts a massive number of checks and controls in place, and require such careful specification and documentation of the process that they can insure that the company will provide an objective finding about whether their drug or device is safe and effective.

FDA is very close on the positivism side of things, but I’ll be honest, it’s really really hard to provide a process that’s not contaminated or influenced by people’s preconceived notions.

Empiricism is a belief that any questions of importance in science can be answered if you collect enough data. I love data. I’m a statistician. I would be in the wrong field if I didn’t love data. But I’ll be damned if I think the data is the be-all and end-all. Data itself has limitations.

How do we collect data? There’s a physical process where you collect data, all right, but there’s also a social process involved and the I strongly believe that how people view things will change how they ask questions in an interview, even if they’re given a script.

You need to think about this because of no one believes positivism to the last degree, but there’s a philosophy called postpositivism which is what we know that there’s going to be biases, but they can be largely removed to the point where we come up with a very close approximation to what is true.

Quantitative research is highly structured, where everything is specified, a priori, and if you deviate from those specifications you have to go to the IRB get an amendment and you have to describe that protocol change in your paper and it’s a ding against you wake comes to critical appraisal.

It’s good to have a well structured protocol. It’s good to specify things a priori to the extent that you can, but I think some of this is naïve and there’s a great quote by Stephen Senn that illustrates this. Stephe Senn is a statistician is done a lot of work with the pharmaceutical industry. So you think you be a positivist. But he said a medical biostatistician is the only one who doesn’t believe in America because it wasn’t in Christopher Columbus’s original research plan.

So the point is that trying to specify everything a priori is naïve.

The goal here of quantitative research and positivism is to separate the researcher from the research.

Quantitative research works with data that can be easily represented as numbers. Now there isn’t anything that can’t be turned into a number if you work hard enough at it, but the process is clear, transparent, and obvious in quantitative research

Quantitative research versus qualitative research

Qualitative research.
- Synonyms(?): Constructivist, humanist.
- Covers five sub-areas: phenomological, grounded theory, ethnographic, case study, and narrative research.
- Open ended questions. Research guides and is guided by the research process.
- Data is open ended text–difficult to represent as numbers
- Measures perceptions, feelings, values
Postpositivism tries to reconcile quantitative and qualitative approaches.

The qualitative/constructivist approach to research rejects the idea of objective collection of data. Biases in the data collection process are inevitable because they are dependent on the paradigm or common way of thinking that researchers share. Rather than separate the researcher from the data collection once the protocol is writen, a qualitative approach inserts the researcher directly into the data collection and explicitly specifies the paradigm that should be used during the research process. The researcher influences the data collection and is simultaneously influenced by the data collection.

The act of collecting data is a social process. So, for example, the U.S. Census makes certain decisions about how they train their census workers where they send them and the instructions that they give to athem. So for example if a knock on the door and no one answers, then they Will try to get information about that house and the inhabitants from the next door neighbor.

That’s a process that the census uses to develop something which is the number of people who live in a certain area. It’s not an objective process is a process is decided based on certain social norms about what’s acceptable and what’s unacceptable.

Constructivism is controversial because it seems to imply that there is no reality, and that everyone constructs their own reality. That’s not true. What ta constructivist believes is that the process in which we decide what’s true and what’s not true is determined by society and society has certain prejudices.

There’s a great book called the The Mismeasure of Man. It’s written by Stephen Jay Gould. Stephen Jay Gould is a famous biologist who did a lot of work with evolution and came up with some really startling changes to the theory of evolution. In the book, The Mismeasure of Man, he looked at how researchers measure intelligence to things like IQ tests and he claims that these measures are flawed because the process of developing an IQ is a social process. People decide what questions go on the IQ test, and what answers represent more intelligence or less intelligence and this is a social process. It is subject to social biases and in particular if the people involved in the development of intelligence testing have either overt or hidden biases, that can create a biased test.

There is also Joel Best, who wrote a book, Damned Lies and Statistics who argues that all statistics are social process and you need to understand the process by which these statistics are derived. That doesn’t mean that you don’t believe the statistics, but you have to first think carefully about how those numbers were created. So for example, there are Criminologists who creates statistics on crime rates. Frequently, these statistics come ultimately from the Police Department. The Police Department is going to provide information about how many crimes occurred where they occurred. The classify the types of crimes and the seriousness of those crimes. Police Departments have certain perspectives on how they want to be seen and this can consciously or unconsciously influence how they collect this data.

Now a constructivist does indeed believe that you cannot separate the researcher from the research. But that doesn’t mean that you don’t do research. What it means is that you understand that the researcher is influenced by the research and influences the research. In particular the researcher should look at the data collection process as it develops and change the process based on what they observe.

So the early data in a constructivist approach or qualitative approach influences how the later data is collected. Now this would be anathema to people who believe that you need to specify everything in advance. But the constructivist says that if you think that produces objective data you’re being naïve. Instead, you should embrace the fact that researchers do influence the research and try to harniess this influence in a structured way.

So, for example, if you run a focus group, you decide what additional questions to ask based on how the earlier questions were answered. To me it seems obvious that you ought to do it that way. If you say I have a script and I’m going to stick with it no matter what, that strikes me as a very naïve approach to research.

Even so, there are times when you want to use a certain questionnaire and you don’t want to deviate. So this idea quantitative is good and qualitative is bad or vice versa is wrong. I want to get you away from that narrow-mindedness. There are times when you want quantitative approach in their times and you want a qualitative approach.

A defining characteristic of the qualitative/constructivist approach is the use of open ended questions that allow the participants to describe things from their own perspective.

There are attempts to reconcile these two philosophies. Postpositivism (don’t you love that word) recognizes that you can’t totally remove the researcher from the data collection process but sees the quantitative philosophy as a way to minimize the unavoidable biases that all researchers have.

Quantitative data is data that is easily reduced to a number or category, while qualitative data is not so easily reduced. This is kind of a squishy definition because you can create categories (themes) from qualitative data, but it takes a lot of effort.

Why would you ever use qualitative data if it takes more effort? Qualitative data offers a richness that cannot often get captured by quantitative data.

The other important issue with respect to qualitative research are the five broad areas that encompass most qualitative research. I will talk in more detail about these categories later in the class, but let me mention that I don’t like categories. Categories are valuable because when you say I’m doing phenomenological research, people understand what you doing. It’s a shorthand that I think has values.

But if you look at the definitions of the various types of qualitative research. How different, for example, is a case study compared to a narrative review.

Qualitative research often uses open-ended questions and sometimes the questions that you ask change depending on what the answers to previous questions were.

The data that you collect in qualitative research is text–words and it’s often difficult to provide a good representation of words in numbers. Qualitative researchers do this and they do it well, but the process by which they do it is anything but easy.

The other important aspect of qualitative research is the measurement of what I would call soft end points. These are things that are don’t have any obvious physical, biological or chemical manifestation. Soft end points involve perceptions, feelings, values and so forth.

The number of red blood cells for exhibits a hard endpoints all right, but the and that tends to lend itself towards quantitative research. But if you think how the children who live with a sibling with chronic disease cope, you’re going to ask a lot of questions of the children, of the parents, you are going to read narratives and so forth. This is going to all be text and you going to have to make sense of this text. It is worth the extra effort because of the richness that qualitative data provides.

Quote in support of quantitative data

“When you can measure what you are speaking about, and express it in numbers, you know something about it, when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind; it may be the beginning of knowledge, but you have scarely, in your thoughts advanced to the stage of science.”
- Lord Kelvin, as quoted at Goodreads

Quote critical of quantitative data

“The individual source of the statistics may easily be the weakest link. Harold Cox tells a story of his life as a young man in India. He quoted some statistics to a Judge, an Englishman, and a very good fellow. His friend said, Cox, when you are a bit older, you will not quote Indian statistics with that assurance.”
- Josiah Stamp, as quoted at Bartleby

Quote critical of quantitative data

“The Government are very keen on amassing statistics—they collect them, add them, raise them to the nth power, take the cube root and prepare wonderful diagrams. But what you must never forget is that every one of those figures comes in the first instance from the chowty dar [chowkidar] (village watchman), who just puts down what he damn pleases.”

That census worker whose coming up with a count of the number of people who live in certain parts of the United States, you think counting would be easy, but it is not.

A lot of subjective choices are involved with the training of the census takers. The interactions that census takers with the people they are collecting data on is quite dependent on the census taker. They try to standardize data collection as much as they can, but the process is so complex that some level of subjectivity has to creep in. Counting is a process that seems objective, but when you involve humans in the process, it changes things. I’m glad that the Census counts people as well as it does, but when you think about the process, you should recognize that quantitative data are not necessarily going to be all that the empiricists and the positivists claim that it is.

There are times (many times) when quantitative data is the best thing in the world to do. But if your research question is addressing perceptions, emotions, feelings, and values, it is really hard to do that in a way that’s truly objective.

If you have to elicit information in a way that involves two human beings there are interactions. If you try to pretend that these interactions are in some sense, unbiased, and it doesn’t matter whether the interviewer is male or female, I’m sorry but you’re wrong.

If I’m interviewed by female researcher and they ask certain questions about my prostate, I might not give her an honest answer. I’m not saying this to sound sexist or anything, but it’s just a fact of life. How men interact with men is different than how men interact with women which is different than how women interact with women which is different than how women interact with women. People with certain racial identity interact with people of the same identity differently than they interact with people of a different racial identity.

So pretending the quantitative data is rigorous is a mistake. I’d like to think otherwise, because I spend most of my time on the quantitative side of the dichotomy. Darn it all! I need to disabuse of this notion.v

This is a point I’ve made several times before and which I will continue to hammer on throughout the course. There’s no one right way to do statistics.

We try to remove of the influence of the researcher on the research. We try to spell out a process with an algorithm that you can follow that can be done repeatedly by different people and yet come up with the same results. But if you’re constructivist you believe the truth is constructed as a social process and this means that these efforts will fail.

Should you throw up your hands and give up? Not really. You have two choices. First you can make all reasonable efforts to produce data that is objective. So blinding, for example, is something you can do that helps a lot to remove people’s biases. Suppose you were doing a study where half the people had lung cancer have to people didn’t and you asked both groups of how much did you smoke in your lifetime.

If you knew that the patient had cancer, you might probe a little bit harder on some questions. “Are you sure you didn’t smoke any cigarettes? Think hard now.” So keeping the evaluators blinded goes a long way towards the post positivist belief that removing many of the biases will get us something that’s a reasonably good approximation of the truth.

Your other choice is to embrace the fact that researchers bring their prejudices and biases into the research. Ask the researchers to involve themselves more directly in the research process by allowing them to deviate when needed from a strict algorithm. Do it in a way where the deviations are structured in a certain way. It’s not quite as easy, but when you let the researcher adapt, you often end up with a richer and fuller set of data.

Imagine the setting, and I use this example over and over again, about how children cope when they have a sibling with chronic disease. You can’t just put a questionnaire in front of someone and then tally up the results. You have to probe, ask questions, encourage exploration of important topics.

That’s going to increase the subjectivity of results, but it’s also going to improve the richness of the responses.

The thing about quantitative approaches is that when you try to reduce people in numbers, sometimes you lose something important.

When you have a research question, you have to make a choice. You can try to make it as objective as possible in a post-positivist perspective. Or you describe a process which recongizes and encourages the researcher to become part of the research in a constructivist perspective.

Break #2

What you have learned
- Quantitative versus qualitative research
What’s coming next
- Other research dichotomies

Research dichotomies

Dichotomies are always wrong.
- Trichotomies.
- Monochotomies.
- Spectrum.
But they are still useful.
- Shorthand for others.
- Guidance for statistical analysis.
- Helpful for critical appraisal.
No “best” level in these dichotomies.
Mixed methods.

If you have not read chapter 1 of the book by Gliner et al, pease do so. I disagree with a lot of the things that they say in that chapter, but they provide a framework that is invaluable.

I know a lot about research but trying to take everything I know, and put it in a logical sequence and an overarching framework–that’s really hard. Your book succeeds here and that it worth a lot.

But I have to be honest with you. I dislike the dichotomies that this book creates. I believe that dichotomies are always wrong. Every dichotomy is a false dichotomy.

Many times, it’s not a dichotomy, it’s a trichotomy. There’s a third choice that is unmentioned.

Other times, a dichotomy is actually a monochotomy. Two different things are just two sides of the same coin.

What you might call a dichotomy may actually be a continuum like a spectrum or gradient. It’s not A versus B, but A versus Z with B, C, D, etc. in betweeen. You can take a continuum and force it into a dichotomoy, but the choice between what goes on the left side versus what goes on the right is purely arbitrary. So, imagine a sunrise on a cloudy day. It starts out dark but gradually gets a bit brighter until eventually is becomes daylight. Where is the dividing line between day and night when you can’t see when, exactly, the sun peeks over the horizon?

But dichotomies, even false dichotomies, are still useful. Classifying research into category A versus category B provides a shorthand that helps others understand your research. It also provides guidance as to what sort of statistical analysis is most appropriate and helps with critical appraisal of the research literature.

There is a tendency for some people to embrace one end of the dichotomy and sneer at anyone who adopts the other end. This is a big big mistake and something that I have very little patience for. A good researcher uses more than one approach to research. To say, for example, that quantitative research is “hard” science and qualitative research is “soft” science is both inaccurate and fails to recognize that both approaches are needed if we want to make any scientific progress.

I want you to have a very broad view of research. I brag a lot of probably shouldn’t, but if there’s one thing I do better than anyone else it is that I have an extreme breadth of knowledge. I’m not a specialist in what any one area, but I know a lot about a lot of different areas. That’s hard to do but I pride myself on that. Part of it is being old, but more of it is because I work really hard to stay current in many different areas of research. I want you to develop that breadth of knowledge so you can choose from a huge buffet of courses.

Let me digress a bit here and talk about some of the narrow-minded thinking that I see in some researchers.

Later in this class, we’ll talk about quasi-experimental designs. The one thing I will say right now is I hate this term “quasi” because it implies a level of inferiority. Briefy, a quasi-experimental design is an approach where the researcher has enough control to use randomization, but decides to forgo randomization because of ethical or practical constraints.

Forgo randomization? Why would you ever do that? There are times when the benefits that randomization provides (removal of the effects of known and unknown confonding variables, the ability to make rigorous probability statements) are outweighed by the harms produced by randomizaiton. Randomization is expensive and places serious limits on your sample size. Randomization introduces a measure of artificiality into your clinic. Most of the time in the clinic you will tell your patients, “This looks like the best option for you” but not with a randomized trial. In a randomized trial, you have to come up with a bland statement about equipoise. and this can color a patient’s outlook. Randomization also requires your patients to consent to a loss of control over their medical treatment. And it’s not just that they are ceding this choice to a trusted expert. They are ceding this choice to the random flip of a coin.

Many times, I have to admit, the benefits of randomization outweigh the harms, and by quite a bit. But there are plenty of settings where the reverse is true. When this happens, you need to be bold in how you work this. Don’t say you had to SETTLE for an inferior quasi-experimental design because of problems to randomization. Instead, say that the quasi-experimental approach is the best approach because it avoids some of the pitfalls associated with randomization. It turns a negative into a positive.

Here’s a more extreme example: historical controls. There is a lot of sneering at historical controls as being a weak form of research. Many times it is, but some research questions almost demand the use of historical controls.

Suppose you are studying a disease that is 100% fatal. In that case, you should dispose of a placebo based control group. No one wants a placebo when facing a certain death. So you put everyone in the treatment group and compare to a well established historic norm of 100% mortality. If the treatment has any effect at all, it will become rapidly obvious.

This also a new movement afoot within FDA called real-world evidence where you now incorporate historical controls into a randomized trial. Whne you are tesing a new intervention, every interventions is different from every other intervention. But, that control group is run identically across multiple studies. Not always, but a lot of times the control group is the same thing same old same standard treatment. These drug companies might accumulate 10 or 20 control groups there all run pretty much the same except that tehy are run at different times. This is a huge source of information. If you can combine this in a way that allows for the extra variation caused by history temporal trends, you can incorporate that information and greatly reduce the concurrent control that you need for the study.

So don’t think narrowly and always choose one side of any of these dichotomies. In particular, there’s nothing wrong with using both ends of a dichotomy in a single research study. This goes by the term “mixed methods.”

Theoretical versus applied

Theoretical: no benefit to patients now.
Applied: potential for immediate benefit.
“Experience by itself teaches nothing… Without theory, experience has no meaning. Without theory, one has no questions to ask. Hence, without theory, there is no learning.”
- W. Edwards Deming, in The New Economics for Industry, Government, Education.

The distinction between theoretical and applied research is in when you expect to see something of value to your patients. Theoretical research has no immediate benefit to your patients, and could potentially never offer direct benefits to your patients.

There’s a tendency of some applied researchers to sneer at theoretical research. Now remember that I don’t want you to sneer. This is terribly wrong as this quote from W. Edwards Deming explains. Theory is what allows us to make sense of our observations of the world.

Without a theoretical framework to support it, applied research is impossible. The other value of theory is that it allows you to generalize to new settings where observations have never been made before. So I’m a big fan of theory, myself.

The other thing I will say is that I got a very theoretical training when I was a graduate student. It was very mathematical and I use almost none of it but I do what I do today. But it still helps a lot because since I understand the theoretical foundations of various statistical methods that I use. I can also learn new methods whole lot faster because I can peek under the hood and see how they work. It helps me understand where they work well and where they don’t work well.

So I’m glad that I got a strong theoretical foundation–a strong mathematical foundation. But let me expand theory a bit. You’re probably familiar with the school Stowers Institute. They have this gorgeous building that I try to visit whenever I can. It is really fancy with exotic wood paneling everywhere. Stowers supports really theoretical research studing animal models like zebrafish. Now why in the hell do we care about zebrafish? The reason we care is because zebrafish is an ideal animal model for understanding certain genetic changes that could not be studied in species that are closer to humans like a lab rat or a dog or a monkey. All this theory helps us understand things like regeneration better that doesn’t have any immediate application except perhaps to the zebrafish themselves. But the more we know about regeneration, the more ideas that we will develop in areas of direct interest, like how to regenerate pancreas cells in patients with diabetes.

Translational versus basic

Basic research: “without thought of practical ends.”
- .
Translational research: transition from “bench to bedside.”
- Called T1 research.
- Next step (T2): transition from bedside to community.
- T3, T4, T5???

I find the term “translational” to be abused a lot, but it helps to understand the original definition. The translational/basic dichotomy is not too much different from the applied/theoretical dichotomy. Basic research is research done mostly in the laboratory and “without thought of practical ends.” Translational research represents a bridge between the laboratory and the real world. You’ll see the phrase “from bench to bedside” used in this context.

Some people subdivide translational research into two types with T1 research representing a bridge from the laboratory to the patient and T2 research representing a bridge from a the carefully controlled clinical trial to actual practice settings in the regular community.

Some people further divide translational research into even more categories, T3, T4, and even T5. If you have a good resource that explains clearly what T5 research is, I’d encourage you to share it with the rest of the class.

Like every other dichotomy, it is a mistake to view one side of the dichotomy as superior to the other.

Laboratory versus field.

Laboratory: controlled setting
- Unnatural.
- Control extraneous variables.
Field setting: in the clinic
Ecologic validity: “the methods, materials and setting of the study must approximate the real-world that is being examined.”
- Source: Wikipedia.

Another closely related dichotomy is laboratory versus field research. Laboratory research is done in a controlled setting and has a greater level of control. The big criticism of laboratory studies is that they are “unnatural” and do not reflect how clinical care is actually provided. In fairness, almost all research has an artificiality associated with it. It’s just that the laboratory is a bit more artificial than field research.

Now laboratory research does not have to be animals or petri dishes. You can study humans in the laboratory. Many exercise studies will put human volunteers on a treadmill and have them wear a mask that controls and measures their oxygen consumption. It’s great in that you can control things like temperature and humidity. When I run, and I run very slowly, I am outdoors and the temperature could be as high as 90 degrees Fahrenheit or as low as 20 degrees Fahrenheit. I run in anything except the really really cold days. And all the environmental variation is an issue. But the artificiality of running indoors on a treadmill with a maks in your face is also an issue.

Field settings provide what the authors describe as “ecologic validity.” This is a term I’d not heard before but the definition is quite simple. The Wikipedia definition is as good as any.

If you’re trying to decide whether to do a laboratory study versus a field study, ask yourself how much do you already know about the intervention. How much research has already been done.

If you have a fairly novel intervention, you might need to study it in a setting where you can control things carefully. Then when you prove it works in a setting with an artificially high level of control, then you can try to see if it also works in a messier, but less artificial setting of a field study.

Experimental versus observational

Experimental
- Researcher chooses the intervention
- Allows for randomization
Observational
- Patient/doctor chooses the intervention
- Groups not under anyone’s control

Experimental research is a setting where the research has the ability to contol who gets what intervention. It usually, but not always, involves randomization.

An observational study is one where the patient and/or the health care professional gets to choose the intervention or settings where what happens to you is not under anyone’s control.

The process of randomization is artificial because the patient gives up a great deal of autonomy. Be thankful because your patients are sacrificing a huge amount of autonomy and is not just to you who may know more about the disease. They are sacrificing their autonomy to the flip of a coin.

Now the control that comes with experimental research and randomization is substantial. You get much much greater control of extraneous variables. All right, they ALMOST disappear. There are still a few issues you have to worry about, but the process of randomization very powerful.

One the flip side, though, randomization is very artificial. If you offer multiple treatment options in a clinic, you almost always outline the risks and benefits and let the patient choose. That is so much different than saying two options have equivalent risk/benefit ratios and we will let a coin flip decide which one is chosen.

Another level of artificiality is the need for informed consent. If you as a researcer choose the intervention for a patient or let a random device choose the intervention, you have to get permission first. This process (informed consent) will lead to some patients refusing, and the ones that remain may no longer be a representative sample of the population you are studying.

The very process of asking for informed consent provides a dispassionate approach that could color how the patient views either intervention.

Finally, asking the research to involve themselves in this complex process of requesting informed consent and then selecting the intervention, often on the basis of a random coin flip takes time and money. Observational studies usually have much lower overhead. This allows you to study a much larger group of patients, sometimes one or two orders of magnitude larger. You have to worry about the influence of uncontrolled variables, but there are ways to adjust for this.

The choice between experimental research versus observational research is a complex one with many trade-offs. But again, be strong and assertive about your choice. Don’t say you are settling for an observational study because an experimental approach is too difficult. Say that the observational approach that you have chosen is the best approach that you have at your disposal because it has fewer of the problems that an experimental approach might have.

Participant report versus researcher observation

Participant report.
- Either written or oral.
- Only practical approach for pain, quality of life.
- Also known as Patient Reported Outcomes (PRO).
Researcher observation.
- Also includes instruments like a heart rate monitor.
- Perceived as more objective.

Another dichotomy in research involves who is providing the data for the research study. If the patient is providing the data (either through an interview or through their writing), you get something “straight from the horse’s mouth.” There’s always the concern that patients might say things to please the researcher or hide things that they might be embarrassed about. Certain things like pain and quality of life can be measured in no other way. Note that a lot of the resources for patient report appears under the acronym PRO (Patient Reported Outcome).

In contrast to a patient report, the physician or other care giver could provide the data. The data could also come from a machine like a blood pressure cuff or a heart rate monitor. Researcher observation is perceived by many to be more objective, but I disagree.

The other big one is stress. We can’t put a stress meter on someone and it turns bright red when they’re having a lot of stress and it turns green when everything’s calm. Maybe there are some ways to measure brain waves, heart rates, or biochemical markers, but these are all imperfect. There’s no good measure of stress yother than patient reports. You have to ask a series of questions that indicate what they are feeling and experiencing right now. You have to do it carefully and we talk a lot about this in the section of reliability and validity. This is especially important for patient reports, but it is also often needed for researcher observation. So for example of a doctor looks to the patient’s skin and tries to assess whether there is some degradation like the scabbing or chafing, you still have to assure that they can be done reliably.

Summary

What you have learned
- Defining Clinical Research Methodology
- Quantitative versus qualitative research
- Other research dichotomies